from collections import Counter #计数
import re
from snownlp import SnowNLP #词频统计
import pandas as pd #格式
import string
from zhon.hanzi import punctuation #中文标点


def sent_process(file_name):
    result = pd.DataFrame(columns=['words', 'count'])
    cn_words = []
    en_words = []
    all_line = ''
    with open(file_name, 'r', encoding='gbk') as f: #读取文本
        for line in f.readlines():
            line = line.strip()
            if not line:
                continue
            all_line += line 
        for sent in SnowNLP(all_line).sentences:
            sent = re.sub('[%s]' % re.escape(string.punctuation), ' ', sent)
            sent = re.sub('[%s]' % re.escape(punctuation), ' ', sent)
            sent = sent.replace('．', ' ')
            if not sent:
                continue
            for item in SnowNLP(sent).words:
                item = item.strip()
                if re.search('[\u4e00-\u9fa5]', item): #中文
                    cn_words.append(item)
                elif re.search('[a-zA-Z]', item): #英文
                    en_words.append(item)
    cn_counter = sorted(Counter(cn_words).items(), key=lambda x: x[1], reverse=True)
    en_counter = sorted(Counter(en_words).items(), key=lambda x: x[1], reverse=True)
    cnt = 0
    for word, count in cn_counter:
        result.loc[cnt] = [word, count]
        cnt += 1
    for word, count in en_counter:
        result.loc[cnt] = [word, count]
        cnt += 1
    result.to_csv(file_name[:-4] + '.csv', index=False)
    print('Process Over: {}'.format(file_name))


if __name__ == '__main__':
    sent_process('1A_01.爱情与金钱.txt')
    sent_process('1A_02.苏格兰玛丽女王.txt')
    sent_process('1A_03.在月亮下面.txt')
    sent_process('1A_04.潘德尔的巫师.txt')
